Papers with pre-trained representations
Reevaluating Argument Component Extraction in Low Resource Settings (D19-61)
Copied to clipboard
| Challenge: | Argument component extraction is a challenging and complex high-level semantic extraction task. |
| Approach: | They propose to use character-level, GloVe, ELMo, and BERT encodings to compare arguments extracted using standard BiLSTM-CRF encoders. |
| Outcome: | The proposed approaches perform better than baselines in higher-level semantic extraction tasks and suggest future improvements. |
An Exploratory Study on Multilingual Quality Estimation (2020.aacl-main)
Copied to clipboard
Shuo Sun, Marina Fomicheva, Frédéric Blain, Vishrav Chaudhary, Ahmed El-Kishky, Adithya Renduchintala, Francisco Guzmán, Lucia Specia
| Challenge: | Existing approaches to predict the quality of machine translation use language-specific models, but they lack labelled data for each language pair. |
| Approach: | They propose to use scores from translation models to estimate quality of machine translations by predicting the quality of a translation at test time. |
| Outcome: | The proposed models outperform single-language models in less balanced quality label distributions and low-resource settings. |
Temporal Effects on Pre-trained Models for Language Processing Tasks (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study shows that language models can be improved as time passes . a number of approaches to solving language tasks have evolved rapidly without a model . |
| Approach: | They examine temporal effects on model performance on downstream language tasks . they also examine the efficacy of two approaches for temporal domain adaptation without human annotations . |
| Outcome: | The proposed methods improve self-labeling and named entity recognition on new data. |
On the Use of External Data for Spoken Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require large labeled datasets to perform . compared to prior work, relative improvements in F1 of up to 16% are found . |
| Approach: | They propose to use self-training, knowledge distillation, and transfer learning to learn SLU models . they compare pipeline and pipeline approaches to find out how to use external data . |
| Outcome: | The proposed models improve performance beyond pre-trained models in resource-constrained settings . the best baseline model is a pipeline approach, while the best performance is achieved by an E2E model. |
On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in neural architectures and pre-trained representations have greatly improved the performance of fully-supervised semantic role labeling (SRL) but there are limitations in the availability of supervised training data. |
| Approach: | They propose to leverage syntactic dependencies to facilitate cross-lingual transfer by annotating predicate-argument structures in text. |
| Outcome: | The proposed model can be extended to other languages with limited training data. |
Pre-Training BERT on Domain Resources for Short Answer Grading (D19-1)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations have achieved state-of-the-art results on multiple downstream NLP tasks by fine-tuning with task-specific data. |
| Approach: | They propose to augment domain-specific data by using labeled short answering grading data for further enhancement of the pre-trained language model. |
| Outcome: | The proposed model can be enhanced by augmenting data from domain-specific resources like textbooks and labeled short answering grading data. |
TempoFormer: A Transformer for Temporally-aware Representations in Change Detection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to model context and time dynamics are slow and prone to overfitting. |
| Approach: | They propose a transformer-based and temporally-aware model for dynamic representation learning that is task-agnostic and trained on inter and intra context dynamics. |
| Outcome: | The proposed model is task-agnostic and can be used as the temporal representation foundation of other models or applied to different transformer-based architectures. |
On the Fragility of Active Learners for Text Classification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Active learning (AL) techniques optimally utilize a labeling budget by iteratively selecting instances that are most valuable for learning. |
| Approach: | They propose to use active learning techniques to iteratively select instances that are most valuable for learning. |
| Outcome: | The proposed framework is used to benchmark active learning techniques for text classification using pre-trained representations. |